Text Mining on Presidential Speeches

Here I conduct a short story of different speeches of different U.S. presidents and nominees based on the data scraped from http://www.presidency.ucsb.edu/. For this project, speeches includes all inaugural addresses of past presidents, nomination speeches of major party candidates, farewell addresses and several public speeches from Donald Trump. The following textual analysises will be based on these speeches and I will have a especial focus on speeches of the current president Donald Trump.

Part1 Topic Modeling

Let’s firstly have a preview of what topics may be included in speeches concerning politics and presidential election. Let’s assume there are 6 topics in the speeches. Then we run LDA over our data and visualize the top-12-most-frequent words in each topic:

From the above graph, we can surely see some patterns. For instance, some words occur frequently in every topic, like “american”, “work”, “freedom” etc., which makes sense because a political speech is all about contents like them. On the other hand, we can see some diversities among topics. Like, topic1 focuses more on “law” while topic2 pays more attention on “peace and security”. So I manually set topic names to each of the topic: “Legislation”, “HomelandSecurity”, “Election”, “America”, “Reform” and “Work”.

Based on the divided topics, we can clearly find some hit words in political speeches. So now we want to find out how the frequencies of some of hit words change over time.

As we can see, they display different patterns although they are all hit topics. For example, the frequencies of “law” and “consititution” seems to be lower than before, which makes sense because laws are more complete than before. “immigration” and “work” occur more often now which also fits the trend. While “economy” and “trade” occur steadily during the course of development.

analysis ## explanation followed by online book frequency prob etc..

Also, we can cluster different inauguration speeches in terms of their topic:

From the figure, we can see that topics of inauguration speeches do have great diversity in terms of different presidents. For example, Donald Trump and John adams, which come from different clusters with far distance. It makes sense because back when John was president, he talked more about the independence of America. But now, topics concerning how to develop the country are mostly mentioned.

Part2 Analyzing Words

Now let’s do some analysis on the number of words in one sentence. First, let’s focus on all the inauguration speeches in each of different parties:

The message is clear that for “Fedralist”, “Whig”, “Democratic-Republican Party”, there are basically no preferences on using short or long sentences. However, for “Democratic” and “Republican”, they seem to prefer using more short words per sentence. Take Donald Trump for instance, the patterns in his different speeches are especially obvious:

We can see that his inauguration speeches seem normal, while his nominee speeches get to show some tend to use short words. What’s more, his public speeches show a apparent preference on using short words. And the length of each of the three types of speeches seems to increase, which may suggest he might sort of control himself at more formal venues. And let’s take a look at what exactly short sentences he said at different types of speeches.

nominee speeches:

inauguration speeches:

public speeches:

We can see that, at nominee and inauguration speeches, the short sentences do not convey much real information, like “thank you”. While in public speeches, short sentences do mean something. It may express his emotion, comments etc.

Based on the above analysis, since short words are some kind of hit in speeches. Now let’s take an especial look on the present president Donald Trump and focus on the frequency of word gruops (bigram). Because unlike single word, this may actually mean something like some specific topics.

First, let’s visualize what bigrams he said frequently in his different speeches:

Through visualizing that, analyzing bigram is helpful for exploratory analyses of the text. For example, if we want to know what word is ahead of “american” most:

We can see it’s Africa American, which is exactly one of his focus in the speeches.

Or, if we are wondering what kinks of policy he mentioned most:

The results also fit the real trend, which are immigration and foreign policies.

One step further, if we want to know what is his attitude or how he describe immigration, we can check the words that are ahead of “immigration”:

The reuslts have no surprise, because he has an extreme attitude about immigration. So no wonder he always emphasized “illegal” when talking about it.

Part3 Sentiment Analysis

Finally, we will implement sentiment analysis on the inauguration speeches and also have an especial look at Donald Trumps emotioms in his different speeches.

I will use “NRC” lexicon to do sentiment analysis. To do that, I quantify the positivity and negtivity of each word by calculating a score for each of them. The following figure shows the positivity of all the inauguration speeches:

From the figure, we can visualize that Zachary Tylor has the highest value of positivity, which may suggest that he was a good speecher and had a great way to persuade people. On the other hand, Warren Harding seems to have the lowest value of positivity. Anyway, the bright side is that none of the past presidents ends up with a negative value, which implies that inauguration speeches are some sort of good way to convey postive attitute and to provide hope and faith.

Now let’s look at the different speeches of Donald Trump. We also visaulize score of speeches in terms of types.

The patterns in the graph are obvious. We can see that Donald Trump has the highest score value in his inauguration speeches. While in nominee and public speeches, the scores are much lower, which may suggest that he is a “outright” guy who would not control his word even in public.

sentiment by sentence

Now we go one step further to analyze his speeches. This time we will analyze his sentiments by sentence. We will calculte the score of each sentence. Also, we will see what exactly are the most positive (negative) sentences he said. And we will compare them with those occur in another nominee’s speeches, Hillary Clinton.

Donald Trump’s most positive sentences:

Donald Trump’s most negtive sentences:

Hillary Clinton’s most positive sentences:

Hillary Clinton’s most negative sentences:

From the above analysis, we can see some interesting patterns. Based on some of the most positive and negative sentences, we can see that Donald Trump is an “outright” or even “extreme” nominee. We can see that some of his bad words directly aim at someone or something. He kind of cannot control himself saying them. On the other hand, for another nomiee in this year, Hillary Clinton, even her top negative words are more of like stating some facts rather than explicitly aiming at someone.